Three Dimensional User Interface Cursor Control

ABSTRACT

A method, including receiving, by a computer executing a non-tactile three dimensional (3D) user interface, a first set of multiple 3D coordinates representing a gesture performed by a user positioned within a field of view of a sensing device coupled to the computer, the first set of 3D coordinates comprising multiple points in a fixed 3D coordinate system local to the sensing device. The first set of multiple 3D coordinates are transformed to a second set of corresponding multiple 3D coordinates in a subjective 3D coordinate system local to the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication 61/420,809, filed Dec. 8, 2010, U.S. Provisional PatentApplication 61/448,670, filed Mar. 3, 2011, and U.S. Provisional PatentApplication 61/538,970, filed Sep. 26, 2011, which are incorporatedherein by reference.

FIELD OF THE INVENTION

This invention relates generally to user interfaces for computerizedsystems, and specifically to user interfaces that are based onthree-dimensional sensing.

BACKGROUND OF THE INVENTION

Many different types of user interface devices and methods are currentlyavailable. Common tactile interface devices include a computer keyboard,a mouse and a joystick. Touch screens detect the presence and locationof a touch by a finger or other object within the display area. Infraredremote controls are widely used, and “wearable” hardware devices havebeen developed, as well, for purposes of remote control.

Computer interfaces based on three-dimensional (3D) sensing of parts ofa user's body have also been proposed. For example, PCT InternationalPublication WO 03/071410, whose disclosure is incorporated herein byreference, describes a gesture recognition system using depth-perceptivesensors. A 3D sensor, typically positioned in a room in proximity to theuser, provides position information, which is used to identify gesturescreated by a body part of interest. The gestures are recognized based onthe shape of the body part and its position and orientation over aninterval. The gesture is classified for determining an input into arelated electronic device.

As another example, U.S. Pat. No. 7,348,963, whose disclosure isincorporated herein by reference, describes an interactive video displaysystem, in which a display screen displays a visual image, and a cameracaptures 3D information regarding an object in an interactive arealocated in front of the display screen. A computer system directs thedisplay screen to change the visual image in response to changes in theobject.

Documents incorporated by reference in the present patent applicationare to be considered an integral part of the application except that tothe extent any terms are defined in these incorporated documents in amanner that conflicts with the definitions made explicitly or implicitlyin the present specification, only the definitions in the presentspecification should be considered.

The description above is presented as a general overview of related artin this field and should not be construed as an admission that any ofthe information it contains constitutes prior art against the presentpatent application.

SUMMARY OF THE INVENTION

There is provided, in accordance with an embodiment of the presentinvention a method, including receiving, by a computer executing anon-tactile three dimensional (3D) user interface, a first set ofmultiple 3D coordinates representing a gesture performed by a userpositioned within a field of view of a sensing device coupled to thecomputer, the first set of 3D coordinates including multiple points in afixed 3D coordinate system local to the sensing device, and transformingthe first set of multiple 3D coordinates to a second set ofcorresponding multiple 3D coordinates in a subjective 3D coordinatesystem local to the user.

There is also provided, in accordance with an embodiment of the presentinvention a method, including receiving, by a computer executing anon-tactile three dimensional (3D) user interface, a set of multiple 3Dcoordinates representing a gesture performed by a limb of a userpositioned within a field of view of a sensing device coupled to thecomputer, the set of 3D coordinates including first multiple pointsmeasured in a fixed 3D coordinate system local to the sensing device,and transforming any of the first multiple points that indicate the limbmoving along a depth axis local to the sensing device to correspondingsecond multiple points along a depth axis local to the user, upon theset of multiple 3D coordinates indicating that a motion of a jointassociated with the limb exceeds a specified threshold.

There is alternatively provided, in accordance with an embodiment of thepresent invention a method, including presenting, by a computerexecuting a non-tactile three dimensional (3D) user interface, a cursorin proximity to one or more items on a display, receiving a set ofmultiple 3D coordinates representing a gesture performed by a body partof a user being positioned within a field of view of a sensing devicecoupled to the computer, calculating a ratio between a first size of thecursor and a second size of the body part, and positioning the cursorresponsively to the received set of multiple coordinates in proportionto the calculated ratio.

There is additionally provided, in accordance with an embodiment of thepresent invention a method, including presenting, by a computerexecuting a non-tactile three dimensional (3D) user interface, aninteractive cursor in proximity to one or more items on a display,receiving a set of multiple 3D coordinates representing a gestureperformed by a user positioned within a field of view of a sensingdevice coupled to the computer, positioning the interactive cursorresponsively to the received set of multiple coordinates, and conveyingfeedback, indicating a proximity of the cursor to the one or moreobjects.

There is further provided, in accordance with an embodiment of thepresent invention an apparatus, including a display, and a computerexecuting a non-tactile three dimensional (3D) user interface, andconfigured to receive a first set of multiple 3D coordinatesrepresenting a gesture performed by a user positioned within a field ofview of a sensing device coupled to the computer, the first set of 3Dcoordinates including multiple points in a fixed 3D coordinate systemlocal to the sensing device, and to transform the first set of multiple3D coordinates to a second set of corresponding multiple 3D coordinatesin a subjective 3D coordinate system local to the user.

There is additionally provided, in accordance with an embodiment of thepresent invention an apparatus, including a display, and a computerexecuting a non-tactile three dimensional (3D) user interface, andconfigured to receive a set of multiple 3D coordinates representing agesture performed by a limb of a user positioned within a field of viewof a sensing device coupled to the computer, the set of 3D coordinatesincluding first multiple points measured in a fixed 3D coordinate systemlocal to the sensing device, and to transform any of the first multiplepoints that indicate the limb moving along a depth axis local to thesensing device to corresponding second multiple points along a depthaxis local to the user, upon the set of multiple 3D coordinatesindicating that a motion of a joint associated with the limb exceeds aspecified threshold.

There is also provided, in accordance with an embodiment of the presentinvention an apparatus, including a display, and a computer executing anon-tactile three dimensional (3D) user interface, and configured topresent a cursor in proximity to one or more items on the display, toreceive a set of multiple 3D coordinates representing a gestureperformed by a body part of a user being positioned within a field ofview of a sensing device coupled to the computer, to calculate a ratiobetween a first size of the cursor and a second size of the body part,and to position the cursor responsively to the received set of multiplecoordinates in proportion to the calculated ratio.

There is alternatively provided, in accordance with an embodiment of thepresent invention an apparatus, including a display, and a computerexecuting a non-tactile three dimensional (3D) user interface, andconfigured to present an interactive cursor in proximity to one or moreitems on the display, to receive a set of multiple 3D coordinatesrepresenting a gesture performed by a user positioned within a field ofview of a sensing device coupled to the computer, to position theinteractive cursor responsively to the received set of multiplecoordinates, and to convey feedback, indicating a proximity of thecursor to the one or more objects.

There is also provided, in accordance with an embodiment of the presentinvention a computer software product including a non-transitorycomputer-readable medium, in which program instructions are stored,which instructions, when read by a computer executing a non-tactilethree dimensional user interface, cause the computer to receive a firstset of multiple 3D coordinates representing a gesture performed by auser positioned within a field of view of a sensing device coupled tothe computer, the first set of 3D coordinates including multiple pointsin a fixed 3D coordinate system local to the sensing device, and totransform the first set of multiple 3D coordinates to a second set ofcorresponding multiple 3D coordinates in a subjective 3D coordinatesystem local to the user.

There is additionally provided, in accordance with an embodiment of thepresent invention a computer software product including a non-transitorycomputer-readable medium, in which program instructions are stored,which instructions, when read by a computer executing a non-tactilethree dimensional user interface, cause the computer to receive a set ofmultiple 3D coordinates representing a gesture performed by a limb of auser positioned within a field of view of a sensing device coupled tothe computer, the set of 3D coordinates including first multiple pointsmeasured in a fixed 3D coordinate system local to the sensing device,and to transform any of the first multiple points that indicate the limbmoving along a depth axis local to the sensing device to correspondingsecond multiple points along a depth axis local to the user, upon theset of multiple 3D coordinates indicating that a motion of a jointassociated with the limb exceeds a specified threshold.

There is further provided, in accordance with an embodiment of thepresent invention a computer software product including a non-transitorycomputer-readable medium, in which program instructions are stored,which instructions, when read by a computer executing a non-tactilethree dimensional user interface, cause the computer to present a cursorin proximity to one or more items on a display, to receive a set ofmultiple 3D coordinates representing a gesture performed by a body partof a user being positioned within a field of view of a sensing devicecoupled to the computer, to calculate a ratio between a first size ofthe cursor and a second size of the body part, and to position thecursor responsively to the received set of multiple coordinates inproportion to the calculated ratio.

There is alternatively provided, in accordance with an embodiment of thepresent invention a computer software product including a non-transitorycomputer-readable medium, in which program instructions are stored,which instructions, when read by a computer executing a non-tactilethree dimensional user interface, cause the computer to present aninteractive cursor in proximity to one or more items on a display, toreceive a set of multiple 3D coordinates representing a gestureperformed by a user positioned within a field of view of a sensingdevice coupled to the computer, to position the interactive cursorresponsively to the received set of multiple coordinates, and to conveyfeedback, indicating a proximity of the cursor to the one or moreobjects.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIG. 1 is a schematic, pictorial illustration of a 3D user interface fora computer system, in accordance with an embodiment of the presentinvention;

FIG. 2 is a flow diagram that schematically illustrates a method forsubjective coordinate transformation, in accordance with an embodimentof the present invention;

FIG. 3 is a schematic side view illustration of a user standing in aroom and interacting with the non-tactile 3D user interface via a 3Dsensing device, in accordance with an embodiment of the presentinvention;

FIG. 4 is a schematic top-down view illustration of the user interactingwith the non-tactile 3D user interface, in accordance with an embodimentof the present invention;

FIG. 5 is a schematic pictorial illustration of the user performing agesture comprising spherical coordinates, in accordance with anembodiment of the present invention;

FIG. 6 is a schematic pictorial illustration of an X′-Y′ plane thatconverges as the user extends a hand closer to the 3D sensing device, inaccordance with an embodiment of the present invention;

FIG. 7 is a schematic side view illustration of a hand performing a Findgesture by pivoting the hand around an associated stationary elbow, inaccordance with an embodiment of the present invention;

FIG. 8 is a schematic side view illustration of the hand performing aTouch gesture while moving the associated elbow, in accordance with anembodiment of the present invention;

FIG. 9 is a schematic pictorial illustration showing shapes of a ringcursor presented on a display, in accordance with an embodiment of thepresent invention;

FIG. 10 is a schematic pictorial illustration showing hand shadowspresented in proximity to the ring cursor, in accordance with anembodiment of the present invention;

FIG. 11 is a flow diagram that schematically illustrates a method forpresenting a proportional hand cursor, in accordance with an embodimentof the present invention;

FIG. 12 is a schematic pictorial illustration of the non-tactile 3D userinterface implementing the proportional hand cursor, in accordance withan embodiment of the present invention;

FIG. 13 is a flow diagram that schematically illustrates a method forvisualizing cursor proximity to a presented item, in accordance with anembodiment of the present invention; and

FIG. 14 is a schematic pictorial illustration of a cursor that visuallyinteracts with items on the display, in accordance with an embodiment ofthe present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

When using a tactile physical input device such as a mouse, a usertypically manipulates the physical device in a two-dimensional planecomprising a horizontal X-axis and a vertical Y-axis. However, wheninteracting with a non-tactile 3D user interface, the user may performgestures in mid-air, and perform the gestures from different positionswithin a field of view of a 3D sensor coupled to the interface.

As the user performs a gesture, the 3D sensor conveys, to thenon-tactile 3D user interface, a signal indicating a first set ofmultiple 3D coordinates representing the gesture. Typically, the 3Dsensor is stationary and the user may be positioned at differentlocations and/or orientations (i.e., relative to the 3D sensor) whileperforming the gesture. The first set of multiple 3D coordinates may beconsidered to be measured in a first coordinate system local to the 3dsensor. The first coordinate system is typically a fixed coordinatesystem, with the 3D sensor defining an origin for a horizontal axis, avertical axis and a depth axis.

Depending on the user's position relative to the 3D sensor, axes of asubjective coordinate system, as perceived by the user, may not alignwith the axes of the fixed coordinate system. Therefore, it may bedifficult to identify a performed gesture from the coordinates measuredby the 3D sensor. For example, the user may perform a “wave” gesture, bymoving a hand along the subjective coordinate system's horizontal axis.However, the 3D sensor's measured coordinates may indicate movementalong the horizontal axis and the depth axis of the sensor's fixedcoordinate system, if the two sets of axes are misaligned.

Embodiments of the present invention provide methods and systems fortransforming between a first fixed coordinate system of a 3D sensorcoupled to a computer and a second subjective coordinate system local tothe user. Since the user may perform gestures from different positions,the subjective coordinate system is typically a “moving” or variablecoordinate system, with the user defining an origin for a horizontalaxis, a vertical axis and a depth axis. Typically, the origin and thedirections of the horizontal, the vertical and the depth axes of themoving coordinate system may change, as the user changes position withinthe 3D sensor's field of view.

In some embodiments, the transformation is based on a position (i.e.,location and orientation) of the user relative to the 3D sensor.Additionally or alternatively, the transformation may allow for theposition of the 3D sensor in relation to room parameters (e.g., the 3Dsensor may be positioned at an angle not parallel to the room's floor).Using embodiments of the present invention described herein, thenon-tactile 3D user interface can simulate the second coordinate systemlocal to the user, and therefore interpret user gestures with greateraccuracy.

While interacting with traditional two-dimensional (2D) user interfaces,the physical devices described supra typically convey tactile feedbackto the user. However, while interacting with a 3D user interface, theuser may perform gestures without engaging any physical device, andtherefore not receive any tactile feedback. Embodiments of the presentinvention provide methods and systems for conveying visual and/or audiofeedback to the user, thereby compensating for the lack of tactilefeedback.

Coordinate transformations described herein can be used by thenon-tactile 3D user Interface when presenting and positioning userinterface elements on a display such as a cursor (as described in detailhereinbelow), a ZoomGrid control scheme, a joystick user interface, ahorizontal bar user interface, and a sessionless pointing userinterface. The ZoomGrid control scheme is described in U.S. ProvisionalPatent Application 61/521,448, filed Aug. 9, 2011, which is incorporatedherein by reference. The joystick user interface and the horizontal baruser interface are described in U.S. patent application Ser. No.13/161,508, filed Jun. 16, 2011, which is incorporated herein byreference. The sessionless pointing user interface is described in U.S.Provisional Patent Application 61/526,692, filed Aug. 24, 2011, which isincorporated herein by reference.

System Description

FIG. 1 is a schematic, pictorial illustration of a non-tactile 3D userinterface 20 (also referred to herein as the user interface) foroperation by a user 22 of a computer 26, in accordance with anembodiment of the present invention. The non-tactile 3D user interfaceis based on a 3D sensing device 24 coupled to the computer, whichcaptures 3D scene information of a scene that includes the body (or atleast a body part, such as hands 30) of the user. Device 24 or aseparate camera (not shown in the figures) may also capture video imagesof the scene. The information captured by device 24 is processed bycomputer 26, which drives a display 28 accordingly.

Computer 26, executing 3D user interface 20, processes data generated bydevice 24 in order to reconstruct a 3D map of user 22. The term “3D map”refers to a set of 3D coordinates measured with reference to a fixed setof axes in space based on device 24. The 3D coordinates represent thesurface of a given object, in this case the user's body. In oneembodiment, device 24 projects a pattern of spots onto the object andcaptures an image of the projected pattern. Computer 26 then computesthe 3D coordinates of points on the surface of the user's body bytriangulation, based on transverse shifts of the spots in the pattern.Methods and devices for this sort of triangulation-based 3D mappingusing a projected pattern are described, for example, in PCTInternational Publications WO 2007/043036, WO 2007/105205 and WO2008/120217, whose disclosures are incorporated herein by reference.Alternatively, interface 20 may use other methods of 3D mapping, usingsingle or multiple cameras or other types of sensors, as are known inthe art.

Computer 26 typically comprises a general-purpose computer processor,which is programmed in software to carry out the functions describedhereinbelow. The software may be downloaded to the processor inelectronic form, over a network, for example, or it may alternatively beprovided on tangible media, such as optical, magnetic, or electronicmemory media. Alternatively or additionally, some or all of thefunctions of the image processor may be implemented in dedicatedhardware, such as a custom or semi-custom integrated circuit or aprogrammable digital signal processor (DSP). Although computer 26 isshown in FIG. 1, by way of example, as a separate unit from sensingdevice 24, some or all of the processing functions of the computer maybe performed by suitable dedicated circuitry within the housing of thesensing device or otherwise associated with the sensing device.

As another alternative, these processing functions may be carried out bya suitable processor that is integrated with display 28 (in a televisionset, for example) or with any other suitable sort of computerizeddevice, such as a game console or media player. The sensing functions ofdevice 24 may likewise be integrated into the computer or othercomputerized apparatus that is to be controlled by the sensor output.

Subjective Coordinate Transformation

As user 22 performs physical gestures, embodiments of the presentinvention combine the methods described hereinbelow to transform a firstset of coordinates in a fixed coordinate system (local to sensing device24) to a second set of coordinates in a subjective coordinate system(local to user 22). The first set of coordinates is based on signalsreceived from 3D sensing device 24 while user 22 performs the gesture.In embodiments of the invention described herein, the fixed coordinatesystem comprises a generally horizontal X-axis, a generally verticalY-axis and a depth Z-axis, the X, Y and Z axes being mutually orthogonalCartesian axes in space, typically with an origin at device 24.

The subjective coordinate system may comprise a generally horizontalX′-axis, a generally vertical Y′-axis and a depth Z′-axis, the axes alsobeing mutually orthogonal Cartesian axes, and having an origin in thevicinity of the user. In some embodiments, the subjective coordinatesystem may use coordinate systems other than those with Cartesian axes,such as spherical or cylindrical coordinate systems.

FIG. 2 is a flow diagram that schematically illustrates a method forsubjective coordinate transformation, in accordance with an embodimentof the present invention. In a receive step 40, computer 26 receives afirst set of multiple 3D coordinates representing a gesture performed byuser 22. The received multiple 3D coordinates typically comprisemultiple points in the fixed coordinate system local to sensing device24. In a transformation step 42, computer 26 transforms the received setof 3D coordinates to a second set of multiple 3D coordinates in thesubjective coordinate system local to user 22.

The coordinate transformations described hereinbelow enable computer 26to interpret the gestures with greater accuracy. The transformations canhelp compensate for differences in the physical positioning of 3Dsensing device 24, and for differences in the location and orientationof user 22 relative to the 3D sensing device. The coordinatetransformations utilized by computer 26 that are described hereinbelowinclude:

-   -   Vertical tilt (vertical axis) correction.    -   Horizontal angle (horizontal axis) correction.    -   Spherical deformation.    -   Identifying a gesture when the dominant motion is along the        depth axis.    -   Automatic learning.    -   Manual calibration.

Additional coordinate transformations that can be utilized by computer26 in embodiments of the present invention include but are not limitedto a parabolic transformation, an elliptic transformation, a Gaussiantransformation, a multi-section planar transformation, a splinetransformation and a nurbs transformation.

FIG. 3 is a schematic side view illustration of user 22 positioned (inthis case standing) in a room 50 and interacting with user interface 20,in accordance with an embodiment of the present invention. In theconfiguration shown in FIG. 2, 3D sensing device 24 is positioned belowdisplay 28, and has a vertical field of view 52. Additionally, 3Dsensing device 24 is tilted slightly upward, resulting in a Z-axis 54that is not parallel to a floor 56. Alternatively, 3D sensing device 24may be positioned above display 28 and tilted downward.

While positioned within field of view 52 and interacting with userinterface 20, the subjective coordinate system comprises a Z′-axis 58that is assumed to be substantially parallel to floor 56. Embodiments ofthe present invention determine a vertical tilt, which comprises avertical angle 60 between Z-axis 54 and Z′-axis 58.

In some embodiments (i.e., in transformation step 42), computer 26determines vertical angle 60 based on vertical reference data collectedduring a pre-calibration step. For example, computer 26 may track andanalyze a pose of user 22, and use the pose as a vertical reference.Additionally or alternatively, computer 26 may use surfaces in room 50,such as floor 56, a ceiling 62 and a wall (not shown) as references.Further alternatively, 3D sensing device 24 may include an accelerometerconfigured to detect vertical angle 60, and convey a signal indicatingthe vertical angle to computer 26. Upon determining vertical angle 60,computer 26 can store the vertical angle to a calibration matrix, whichcan be used to transform coordinates on Z-axis 54 to coordinates onZ′-axis 58.

FIG. 4 is a schematic top view of user 22 interacting with 3D userinterface 20, in accordance with an embodiment of the present invention.In the configuration shown in FIG. 4, user 22 is standing off to theside of display 28 and 3D sensing device 24, but within a horizontalfield of view 70. Since user 22 is not standing directly in front of 3Dsensing device 24, the subjective coordinate system may use a Z′-axis72, comprising a line between the user and device 24, while the fixedcoordinate system may use Z-axis 54, defined, as explained above, by theposition of device 24.

If 3D sensing device 24 and display 28 are substantially alignedhorizontally, and horizontal field of view 70 is known, then computer 26can determine a horizontal angle 76 between Z′-axis 72 and Z-axis 54, bymethods generally similar to those described with reference to FIG. 3.Computer 26 can store horizontal angle 76 to the calibration matrix,thereby enabling the user interface to transform coordinates measuredwith respect to Z-axis 54 to coordinates measured with respect toZ′-axis 72.

The description above illustrates that, in general, values of verticalangle 60 and horizontal angle 76 enable transformation between the fixeddepth axis defined with respect to device 24 and the subjective depthaxis defined with respect to user 22.

FIG. 5 is a schematic pictorial illustration of user 22 performing agesture, which may be conveniently defined as a series of sphericalcoordinates, in accordance with an embodiment of the present invention.Since user 22 does not interact tangibly with a physical surface of theuser interface, the gesture may be more efficiently analyzed using asubjective spherical coordinate system local to the user.

As user 22 performs gestures with hand 30, the actual motion of the handis typically influenced by various joints in the user's body, includingwrist 80, elbow 82 and shoulder 84. For example, while a wave gesturemay be defined as moving hand back and forth horizontally (i.e.,side-to-side), the hand may actually move along an arc 86 due to arotation of the joints. Likewise, while performing a push focus gestureor a touch gesture (described in further detail hereinbelow), where hand30 moves forward and backward relative to user 22, the direction of theforward and the backward motion may depend on the location andorientation of the user (i.e., relative to sensing device 24), and thelocation of hand 30 along arc 86. The forward and backward motion istypically substantially perpendicular to arc 86, as indicated by arrows88. The push focus gesture and the wave gesture are described in U.S.Provisional Patent Application 61/422,239, filed on Dec. 13, 2010, whichis incorporated herein by reference.

While the example in FIG. 5 shows user 22 performing a horizontal Findgesture (i.e., by moving hand 30 from either left to right or right toleft), the user may also perform the Find gesture by moving the hand ina vertical (i.e., up or down) or in a diagonal direction. Therefore, arc86 may comprise an arc along a horizontal plane (as shown in FIG. 5), avertical plane, or a diagonal plane.

As user 22 performs a gesture, embodiments of the present inventionenable computer 26 to transform a first set of Cartesian coordinatesreceived from sensing device 24 to a second set of spherical coordinatesin a subjective spherical coordinate system local to user 22.Transforming the Cartesian coordinates to the spherical coordinates isreferred to herein as a spherical deformation.

In some embodiments, the spherical deformation typically involvescomputer 26 measuring spherical coordinates along a non-linear arc 86with respect to a reference point 90, where the reference pointcomprises a center of arc 86, in proximity to user 22. In someembodiments, since reference point 90 comprises the center of arc 86,coordinates on the arc may be assumed to be substantially equidistantfrom the reference point.

In additional embodiments, computer 26 may select reference point 90based on the user's physical dimensions and current pose (e.g.,standing, sitting or leaning back), since the location of referencepoint 90 may depend on the particular body joint performing the majorityof the motion during a gesture. For example, if user 22 is standing,computer 26 may set reference point 90 to be a location corresponding toshoulder 84. Alternatively, computer 26 may set reference point 90 to bea location corresponding to elbow 82 when the user is sitting.Additionally, computer 26 may adjust the distance that hand 30 needs tomove while performing a gesture based on the user's current pose. Forexample, while performing a side-to-side wave gesture, user 22 may movehand 30 a greater distance when standing than when sitting.

In alternative embodiments, upon user 22 performing a push focusgesture, computer 26 may select reference point 90 as a specificdistance 92 (typically between 50 and 70 centimeters) behind the user.After selecting reference point 90, computer can calculate a radius 94as a distance measured between reference point 90 and a location whereuser 22 extended hand 30 while performing the push focus gesture.

In further embodiments, computer 26 may store parameters such asreference point 90 and radius 94 as spherical deformation parameters fora spherical function configured to transform the first set of Cartesiancoordinates to the second set of spherical coordinates.

FIG. 6 is a schematic pictorial illustration of hand 30 moving forwardand backward along a depth Z′-axis 102 local to the user, in accordancewith an embodiment of the present invention. Although user 22 may intendto move hand 30 solely along Z′-axis 102, the user may alsounintentionally move the hand in an X′-Y′ plane 100, which comprises ahorizontal X′-axis 104 and a vertical Y′-axis 106. X′-Y′ plane 100,X′-axis 104 and Y′-axis 106 are also local to user 22.

To interact with user interface 20, user 22 typically performs gesturesby moving hand 30 along X′-Y′ plane 10 and Z′-axis 102. Whileinteracting with user interface 20, user 22 may manipulate items 108,110 and 112 that computer 26 presents on screen 28. In some embodiments,items 108, 110 and 112 may comprise icons and buttons similar to thosefound on traditional two dimensional (2D) user interfaces. Examples ofgestures performed by user 22 include:

-   -   A “Find” gesture comprises moving hand 30 along X′-Y′ plane 100        in order to highlight, using a cursor 114, item 110. In the        example shown in FIG. 5, in response to the Find gesture        computer 26 highlights item 110 by presenting a shadow around        item 110.    -   A “Touch” gesture comprises moving hand 30 forward along Z′-axis        102, thereby selecting item 110.

In operation, user 22 typically manipulates cursor 114 by moving hand 30along an X′-Y′ plane 100 until the cursor passes over and highlightsitem 110. The manipulation corresponds to the Find gesture describedabove. Once computer 26 highlights item 110, user 22 can move hand 30forward along Z′-axis 102 to a reference touch point 116, thereby“touching” (i.e., selecting) the highlighted item. However, while movinghand 30 along Z′-axis 102, user 22 may also unintentionally move hand 30along X′-Y′ plane 100.

As shown in FIG. 6, computer 26 “converges” X′-Y′ plane 100 by reducingthe impact of any motion detected in the X′-Y′ plane, once the dominantcomponent of the motion (i.e., of hand 30) is along Z′-axis 102. Inother words, as hand 30 transitions from the Find gesture to the Touchgesture, computer assigns less significance to any motion along X′-Y′plane 100.

When user 22 performs a Find gesture followed by a Touch gesture,computer 26 may detect a slight pause as hand 30 changes direction(i.e., from a side-to-side motion to a forward and backward motion). Insome embodiments, computer 26 may assign less significance to any motionin X′-Y′ plane 100, upon detecting a slight pause between motionprimarily in X′-Y′ plane 100 and motion primarily along Z′-axis 102.Additionally, computer 26 may reposition reference touch point 116 to aspecific distance from a location where hand 30 transitioned from theFind gesture to the Touch gesture.

In some embodiments, computer 26 may employ an automatic learningalgorithm to personalize user interface 20 to user 22. By continuallyobserving the motion of hand 30, computer 26 can employ the automaticlearning algorithm to estimate 3D coordinates of the gestures withgreater accuracy. Additionally, since the majority of the hand-motion istypically in X′-Y′ plane 100, the learning algorithm can employstatistical methods to estimate any coordinate rotation along the X′-Y′plane.

For example, when user 22 performs a push focus gesture, the automaticlearning algorithm can calibrate computer 26 to the user's local Z′-axisbased on the direction of the push focus gesture. Additionally oralternatively, if user 22 performs a wave gesture, then the automaticlearning algorithm can calibrate computer 26 to the user's localX′-axis.

In further embodiments, computer 26 may prompt user 22 to performspecific gestures that manually calibrate the user interface to theuser's local X′-axis, Y′-axis and Z′-axis. Examples of manualcalibration operations include

-   -   Raising hand 30. By moving hand 30 in an up-and-down motion,        computer 26 can calibrate the Y′-axis for user 22.    -   Four corner pushes. By moving hand 30, user 22 can first        position cursor 114 at each “corner” of display 28 and then        perform a push focus gesture. Each of the push focus gestures        provides computer 26 with a separate Z′-axis local to its        respective corner. Computer 26 can then employ a compensation        algorithm that interpolates a Z′-axis direction estimation for        each coordinate on display 28 corresponding to a location of        hand 30 within the four “corners”.    -   Rectangle gesture. By moving hand 30, user 22 can position        cursor 114 to “draw” a work surface rectangle on display 28,        thereby aligning X′-Y′ plane 100 to the plane of the rectangle        drawn by the user.    -   Circle gesture. When user 22 performs a circle gesture, computer        26 can align X′-Y′ plane 100 to the plane of the circle drawn by        the user.    -   Virtual surface rubbing. When user 22 performs a “rubbing”        gesture, computer 26 can define a work surface, by optimizing        its geometry to process coordinates collected during the motion.        During the “rubbing,” computer 26 can collect multiple        coordinates of hand 30 during a defined period. Using the        collected coordinates, computer 26 can optimize the parameters        of a linear surface that best fits the collected coordinates,        thereby aligning X′-Y′ plane 100 to the plane of the surface        “rubbed” by user 22.

As discussed supra, computer 26 employs vertical axis coordinatetransformations, horizontal axis coordinate transformations, sphericalcoordinate transformations, and X′-Y′ plane 100 convergence in order totransform a first set of 3D coordinates in a fixed 3D coordinate systemlocal to sensing device 24 to a second set of corresponding multiple 3Dcoordinates in a subjective 3D coordinate system local to the user. Inoperation, computer 26 may utilize one or more of the transformationsdescribed hereinabove.

In some embodiments computer 26 may execute the aforementionedtransformations sequentially as follows:

-   -   Sensing device 24 conveys a signal to computer 26 indicating        multiple 3D coordinates representing a gesture performed by user        22, where the first set of 3D coordinates comprising multiple        points in a fixed 3D coordinate system local to sensing device        24.    -   Computer 26 performs a vertical axis transformation by applying        the calibration matrix to transform the coordinates from Z-axis        54 to coordinates on Z′-axis 58.    -   Computer 26 performs a horizontal axis transformation by        applying the calibration matrix to transform the coordinates        from Z-axis 74 to coordinates on Z′-axis 72.    -   Computer 26 transforms the first set of Cartesian coordinates        received from sensing device 24 to a second set of spherical        coordinates in the subjective spherical coordinate system local        to user 22.    -   Computer 26 assigns less significance to any motion along X′-Y′        plane 100, upon determining that the dominant component of the        motion (i.e., of hand 30) is along Z′-axis 102.

While performing the Find gesture, user 22 may unintentionally move hand30 along the depth Z′-axis local to the user. For example, if the useris sitting on a chair, rests an elbow on the chair's armrest, andperforms an up-and-down Find gesture, the hand will typically move alongarc 86 (that includes local Z-axis coordinates) due to a rotation of theelbow joint.

In some embodiments, computer 26 performs a subjective Z-axis coordinatetransformation by differentiating between intentional and unintentionalmotion of the hand 30 along the depth Z′-axis local to the user by usingassociated elbow 82 (i.e., the right elbow when moving the right hand,and the left elbow when moving the left hand) of the user as a referencewhen determining the significance of the hand's motion on the localZ′-axis.

For example, if user 22 keeps elbow 82 relatively stationary whileperforming a Find gesture, then computer 26 may assign less significanceto the hand's motion detected on the local Z′-axis. However, if user 22moves elbow 82 while performing the Touch gesture, then computer 26 mayassign greater significance to the hand's motion detected on the localZ′-axis. In other words, if user 22 raises elbow 82 while moving hand 30forward, then embodiments of the present invention can assume that theuser is intentionally moving the hand forward along the local Z′-axis.

FIG. 7 is a schematic side view illustration of hand 30 performing aFind gesture, in accordance with an embodiment of the present invention.In the example shown in FIG. 7, user performs the Find gesture bypivoting hand 30 around associated elbow 82, which is resting (i.e.,stationary) on a surface 118. As user 22 moves hand 30 along arc 86 andwithin field of view 52 of 3D sensing device 24, computer 26 receives afirst signal from the 3D sensing device indicating a first set ofmultiple 3D coordinates representing the Find gesture, where the firstset of 3D coordinates comprises multiple points measured relative todevice 24.

While hand 30 performs a Find gesture, computer 26 receives a signalfrom 3D sensing device 24 indicating a motion performed by hand 30 andits associated elbow 82, where the signal comprises a multiple 3Dcoordinates indicating first multiple points in the subjectivecoordinate system. If the received multiple 3D coordinates indicatemovement of elbow 82 along Z′-axis 58 to be within a specified threshold(e.g., two centimeters) while hand 30 moves along arc 86, then thecomputer may assign less significance to the hand's motion along Z′-axis58. In other words, as user 22 performs a Find gesture (i.e., and themotion of elbow 82 is within the specified threshold), computer 26 maynot transform the first multiple points in the first set of 3Dcoordinates indicating hand 30 moving along Z-axis 54 to correspondingsecond multiple points on Z′-axis 58.

FIG. 8 is a schematic side view illustration of hand 30 performing aTouch gesture while moving associated elbow 82, in accordance with anembodiment of the present invention. As user moves hand 30 forward alongZ′-axis 58 and within field of view 52, computer 26 receives the firstsignal from the 3D sensing device indicating the first set of multiple3D coordinates representing the Touch gesture, where the first set ofmultiple 3D coordinates comprises multiple points on Z-axis 54.

While hand 30 performs a Touch gesture, computer 26 receives a signalfrom 3D sensing device 24 indicating a motion performed by hand 30 andits associated elbow 82, where the signal comprises multiple 3Dcoordinates indicating first multiple points in the subjectivecoordinate system. If the received multiple 3D coordinates indicatemovement of elbow 82 along Z′-axis 58 to be greater than the specifiedthreshold while hand 30 moves along arc 86, then the computer may assigngreater significance to the hand's motion along Z′-axis 58. In otherwords, as user 22 performs a Touch gesture (i.e., and the motion ofelbow 82 is greater the specified threshold), computer 26 may transformthe first multiple points in the first set of 3D coordinates indicatinghand 30 moving along Z-axis 54 to corresponding second multiple pointson Z′-axis 58.

Although the examples shown in FIG. 8 show user 30 moving hand 30 andelbow 82 while performing the Find gesture, alternative gestures usingdifferent limb and joint combinations are considered to be within thespirit and scope of the present invention. For example, computer 26 candifferentiate between intentional and unintentional motion of the user'sfoot (not shown) along the subjective Z′-axis to the user by using theuser's knee (not shown) as a reference when determining the significanceof the foot's motion on the local Z′-axis.

Cursor Visualization

Although user 22 may perform three-dimensional gestures whileinteracting with user interface 20, visual feedback is typicallypresented in two dimensions on display 28. Conveying effective visualand/or audio feedback to user 22 enhances the ability of the user tointeract with user interface 20. Embodiments of the present inventiondescribed hereinbelow present various cursor configurations on display28 that convey visual feedback to user 22. In some embodiments, computer26 may convey audio feedback to user 22 while the user interacts withthe user interface.

As described supra, computer 26 receives signals from 3D sensing device24 indicating a first set of coordinates representing a gestureperformed by user 22, using hand 30. Upon receiving the first set ofcoordinates, which are in a fixed coordinate system local to sensingdevice 24, computer 26 may then transform the received coordinates to asecond set of coordinates that are in a moving coordinate system localto user 22. Embodiments of the present invention convey feedbackindicating the hand's proximity to reference touch point 116 along depthZ′-axis 102 in the coordinate system local to user (FIG. 6). Since theuser interface is three dimensional, interacting with items presented ondisplay 28 typically comprises hand 30 moving on Z′-axis 102 (e.g., theTouch gesture discussed supra).

FIG. 9 is a schematic pictorial illustration showing variousconfigurations of a ring cursor 120 presented on display 28, inaccordance with an embodiment of the present invention. Ring cursor 120conveys visual feedback to user 22 as to a position of hand 30 onZ′-axis 102 relative to reference touch point 116. A scale 124 indicatesthe distance between hand 30 and reference touch point 116. As user 22moves hand 30 towards reference touch point 116 along Z′-axis 102,computer 26 may proportionally fill in ring 120, making the ring thickeruntil completely filling in the ring at the point where the hand reachesthe reference touch point.

Upon hand 30 reaching reference touch point 116, computer may present avisual effect, such as flashing different colors within ring cursor 120,thereby enabling user 22 to realize that the hand has reached thereference touch point. Likewise, as user 22 retracts hand 30 back awayfrom reference touch point 116, computer 26 may gray out the ringcursor.

In some embodiments, computer 26 may change the configuration of ringcursor 120 as user 22 extends hand 30 beyond reference touch point 116.In the configuration shown in FIG. 9, computer 26 adds a second innerring 122 to the ring cursor, upon hand 30 reaching reference touch point116. If user 30 extends hand 30 beyond reference touch point 116 (e.g.,−5 centimeters on scale 124), the user interface may change the size ofinner ring 122.

FIG. 10 is a schematic pictorial illustration showing a hand shadow oricon 130 displayed in proximity to ring cursor 120, in accordance withan embodiment of the present invention. The proximity of shadow 130 toring cursor 120 may depend on the proximity of hand 30 to referencetouch point 116. Additionally, as hand 30 reaches reference touch point116 on Z′-axis 102, computer 26 may simulate the hand pressing a buttonby presenting fingers in shadow 130 being slightly bent back.

FIG. 11 is a flow diagram that schematically illustrates a method forpresenting a proportional hand cursor 150, in accordance with anembodiment of the present invention, and FIG. 12 is a schematicpictorial illustration of user interface 20 implementing theproportional hand cursor, in accordance with an embodiment of thepresent invention. In some embodiments, computer 26 may present handcursor 150 as proportional to the size of hand 30 as measured by device24. Implementing proportional cursor 150 can help user 22 acclimate tothe non-tactile 3D user interface, since computer 26 can present themotion of hand cursor 150 as proportional to the motion of hand 30.

In a presentation step 140, computer 26 presents hand cursor 150 ondisplay 28, where the hand cursor is in proximity to one or more items(e.g., icons) also presented on the display. In a receive step 142,computer 26 receives a set of multiple 3D coordinates representing agesture that user 22 performs using hand 30, and in a calculation step144, the computer calculates a ratio between the size of hand cursor 150and the measured size of the hand. In a presentation step 146, computer26 positions hand cursor responsively to the received coordinates inproportion to the calculated ratio, and the method continues with step142.

In the example shown in FIG. 12, hand cursor 150 and a button icon 152are separated by a distance 154 on display 28. Distance 154 isapproximately three times the width of hand cursor 150. Therefore, inpresentation step 146, computer 26 can present hand cursor 150 movingproportionally to the movement of hand 30. In other words, as user 22moves hand 30 a distance 156, computer 26 moves hand cursor 150 bydistance 154, where distance 154 comprises three times the width of thehand cursor.

In some embodiments, computer 26 can present hand cursor 150 as asemi-realistic hand, and change the hand cursor's appearance as user 22moves hand 30 along Z′-axis 102. By changing the appearance of handcursor 150, computer 26 can convey visual feedback to user 22. Forexample, as user 22 performs a Touch gesture to “push” button icon 152,computer 26 can present hand cursor 150 with one or more fingers bentback.

In an alternative embodiment, computer 26 can present a combination ofring cursor 120 and hand cursor 150 on display 28. Hand cursor 150 canprovide depth feedback by altering the size of the hand cursor based onthe position of hand 30 along Z′ axis 102, and ring cursor 120 canprovide feedback to the hand's location on X′-Y′ plane 100. Maintainingthe size of ring cursor 120 constant conveys feedback while the userpositions hand 30 on X′-Y′ plane 100 (e.g., positioning the hand toengage a specific icon). Alternatively, user interface can presentvisual effects such as crosshairs in the ring cursor or glowingfingertips in the hand cursor to help user 30 position hand 30 on X′-Y′plane 100.

While FIGS. 11 and 12 illustrate computer 26 presenting hand cursor 150in response to a gesture that user 22 performs with hand 30, other types(i.e., shapes) of cursors presented by computer 26 and representative ofhand 30 are considered to be within the spirit and scope of the presentinvention. Additionally or alternatively, computer 26 may proportionallyposition cursor 150 in response to user 22 moving a different body part,such as a foot (not shown).

FIG. 13 is a flow diagram that schematically illustrates a method forvisualizing interaction of an interactive cursor 170 with givenpresented items 172, in accordance with an embodiment of the presentinvention, and FIG. 14 is a schematic pictorial illustration of cursor170 interacting with one of items 172, in accordance with an embodimentof the present invention. The interaction between cursor 170 and items172 helps compensate for the lack of tactile feedback in user interface20.

In a presentation step 160, computer 26 presents interactive cursor 170in proximity to items 172. In a receive step 162, computer 26 receives aset of 3D coordinates representing a gesture performed by user 22, andin a presentation step 164, the computer presents interactive cursor 170responsively to the received coordinates. In a convey step 166, computer26 conveys, via interactive cursor 170, feedback indicating theinteractive cursor's proximity to a particular item 172, and the methodcontinues with step 162.

In a feedback step 166, computer 26 conveys feedback (e.g., visualfeedback) indicating the interactive cursor's interaction with items172. In the example shown in FIG. 14, computer 26 can change the shapeof cursor 170 by presenting cursor 170 as a “blob” (i.e., a water-droplike shape) that the computer can deform when moving the cursor overitems 172. Additional examples of visual feedback (i.e., interaction)that computer 26 may present via interactive cursor 170 include:

-   -   Computer 26 may present cursor 170 as a hand, similar in        appearance to hand cursor 150. As computer 26 moves cursor 170        across items 172 (in response to user 22 moving hand 30 in X′-Y′        plane 100), the computer can present fingers of the cursor that        move in a manner similar to fingers of hand 30 playing a piano.    -   Computer 26 may present cursor 170 as a geometric shape, such as        a cube, a cone, a triangle, or an arrow. As computer 26        positions cursor 170 near items 172, the user interface can        “tilt” the geometric shape towards the closest item.

In addition to visual feedback, computer 26 can convey audio feedback touser 22 while the user interacts with the user interface. Conveyingaudio feedback can help compensate for the lack of tactile feedback inuser interface 20. Examples of audio feedback include:

-   -   Computer 26 can convey a low frequency clicking sound when hand        30 “touches” an item (e.g., one of items 172). The audio        feedback can help user 22 interact with user interface 20, even        when the user is focusing on other screen elements, or engaging        in a conversation.    -   Computer 26 can convey a low-volume hissing sound when hand 30        is in proximity on Z′-axis 102 to reference touch point 116.        Computer 26 can adjust the volume of the hissing sound in        proportion to hand 30's distance from the reference touch point.        For example, as hand 30 moves along Z′-axis 102 towards        reference touch point 116, computer 26 can increase the volume        of the hissing sound, and vice versa. Additionally or        alternatively, computer 26 can convey the hissing sound as hand        30 moves along X′-Y′ plane 100, thereby mimicking a sound        resulting from hand 30 rubbing on a physical surface.

It will be appreciated that the embodiments described above are cited byway of example, and that the present invention is not limited to whathas been particularly shown and described hereinabove. Rather, the scopeof the present invention includes both combinations and subcombinationsof the various features, including the transformations and themanipulations, described hereinabove, as well as variations andmodifications thereof which would occur to persons skilled in the artupon reading the foregoing description and which are not disclosed inthe prior art.

1-16. (canceled)
 17. A method, comprising: receiving, by a computerexecuting a non-tactile three dimensional (3D) user interface, a set ofmultiple 3D coordinates representing a gesture performed by a limb of auser positioned within a field of view of a sensing device coupled tothe computer, the set of 3D coordinates comprising first multiple pointsmeasured in a fixed 3D coordinate system local to the sensing device;and transforming any of the first multiple points that indicate the limbmoving along a depth axis local to the sensing device to correspondingsecond multiple points along a depth axis local to the user, upon theset of multiple 3D coordinates indicating that a motion of a jointassociated with the limb exceeds a specified threshold.
 18. The methodaccording to claim 17, wherein the body part comprises a hand and thejoint comprises an elbow associated with the hand.
 19. A method,comprising: presenting, by a computer executing a non-tactile threedimensional (3D) user interface, a cursor in proximity to one or moreitems on a display; receiving a set of multiple 3D coordinatesrepresenting a gesture performed by a body part of a user beingpositioned within a field of view of a sensing device coupled to thecomputer; calculating a ratio between a first size of the cursor and asecond size of the body part; and positioning the cursor responsively tothe received set of multiple coordinates in proportion to the calculatedratio.
 20. The method according to claim 19, wherein the body partcomprises a hand.
 21. A method, comprising: presenting, by a computerexecuting a non-tactile three dimensional (3D) user interface, aninteractive cursor in proximity to one or more items on a display;receiving a set of multiple 3D coordinates representing a gestureperformed by a user positioned within a field of view of a sensingdevice coupled to the computer; positioning the interactive cursorresponsively to the received set of multiple coordinates; and conveyingfeedback, indicating a proximity of the cursor to the one or moreobjects.
 22. The method according to claim 21, wherein the feedbackcomprises a sound.
 23. The method according to claim 21, wherein thefeedback comprises changing a shape of the cursor. 24-39. (canceled) 40.An apparatus, comprising: a display; and a computer executing anon-tactile three dimensional (3D) user interface, and configured toreceive a set of multiple 3D coordinates representing a gestureperformed by a limb of a user positioned within a field of view of asensing device coupled to the computer, the set of 3D coordinatescomprising first multiple points measured in a fixed 3D coordinatesystem local to the sensing device, and to transform any of the firstmultiple points that indicate the limb moving along a depth axis localto the sensing device to corresponding second multiple points along adepth axis local to the user, upon the set of multiple 3D coordinatesindicating that a motion of a joint associated with the limb exceeds aspecified threshold.
 41. The apparatus according to claim 40, whereinthe body part comprises a hand and the joint comprises an elbowassociated with the hand.
 42. An apparatus, comprising: a display; and acomputer executing a non-tactile three dimensional (3D) user interface,and configured to present a cursor in proximity to one or more items onthe display, to receive a set of multiple 3D coordinates representing agesture performed by a body part of a user being positioned within afield of view of a sensing device coupled to the computer, to calculatea ratio between a first size of the cursor and a second size of the bodypart, and to position the cursor responsively to the received set ofmultiple coordinates in proportion to the calculated ratio.
 43. Theapparatus according to claim 42, wherein the body part comprises a hand.44. An apparatus, comprising: a display; and a computer executing anon-tactile three dimensional (3D) user interface, and configured topresent an interactive cursor in proximity to one or more items on thedisplay, to receive a set of multiple 3D coordinates representing agesture performed by a user positioned within a field of view of asensing device coupled to the computer, to position the interactivecursor responsively to the received set of multiple coordinates, and toconvey feedback, indicating a proximity of the cursor to the one or moreobjects.
 45. The apparatus according to claim 44, wherein the feedbackcomprises a sound.
 46. The apparatus according to claim 44, wherein thefeedback comprises the computer changing a shape of the cursor. 47.(canceled)
 48. A computer software product comprising a non-transitorycomputer-readable medium, in which program instructions are stored,which instructions, when read by a computer executing a non-tactilethree dimensional user interface, cause the computer to receive a set ofmultiple 3D coordinates representing a gesture performed by a limb of auser positioned within a field of view of a sensing device coupled tothe computer, the set of 3D coordinates comprising first multiple pointsmeasured in a fixed 3D coordinate system local to the sensing device,and to transform any of the first multiple points that indicate the limbmoving along a depth axis local to the sensing device to correspondingsecond multiple points along a depth axis local to the user, upon theset of multiple 3D coordinates indicating that a motion of a jointassociated with the limb exceeds a specified threshold.
 49. A computersoftware product comprising a non-transitory computer-readable medium,in which program instructions are stored, which instructions, when readby a computer executing a non-tactile three dimensional user interface,cause the computer to present a cursor in proximity to one or more itemson a display, to receive a set of multiple 3D coordinates representing agesture performed by a body part of a user being positioned within afield of view of a sensing device coupled to the computer, to calculatea ratio between a first size of the cursor and a second size of the bodypart, and to position the cursor responsively to the received set ofmultiple coordinates in proportion to the calculated ratio.
 50. Acomputer software product comprising a non-transitory computer-readablemedium, in which program instructions are stored, which instructions,when read by a computer executing a non-tactile three dimensional userinterface, cause the computer to present an interactive cursor inproximity to one or more items on a display, to receive a set ofmultiple 3D coordinates representing a gesture performed by a userpositioned within a field of view of a sensing device coupled to thecomputer, to position the interactive cursor responsively to thereceived set of multiple coordinates, and to convey feedback, indicatinga proximity of the cursor to the one or more objects.